Online Adaptive Decision Fusion Framework Based 
on Entropic Projections onto Convex Sets with 
Application to Wildfire Detection in Video 

Osman Giinay*, Behcet Ugur Toreyin^, Kivanc Kose* and A. Enis £etin* 
*Department of Electrical and Electronics Engineering 

Bilkent University, Ankara, Turkey, 06800 
Telephone: +90-312-290-1219 Fax: +90-312-266-4192 
Email: {osman,kkivanc,cetin}@ee.bilkent.edu.tr 
^Texas A&M University at Qatar, Education City, 23874, Doha, Qatar 
Telephone:+974-5508-9336 
Email: behcet.toreyin@qatar.tamu.edu 



(EDICS:) ARS-IIU:Image & Video Interpretation and 
Understanding Object recognition and classification; Fore- 
ground/background segregation; Scene analysis 

Abstract — In this paper, an Entropy functional based online 
Adaptive Decision Fusion (EADF) framework is developed for 
image analysis and computer vision applications. In this frame- 
work, it is assumed that the compound algorithm consists of 
several sub-algorithms each of which yielding its own decision as 
a real number centered around zero, representing the confidence 
level of that particular sub-algorithm. Decision values are linearly 
combined with weights which are updated online according to an 
active fusion method based on performing entropic projections 
onto convex sets describing sub-algorithms. It is assumed that 
there is an oracle, who is usually a human operator, providing 
feedback to the decision fusion method. A video based wildfire 
detection system is developed to evaluate the performance of the 
algorithm in handling the problems where data arrives sequen- 
tially. In this case, the oracle is the security guard of the forest 
lookout tower verifying the decision of the combined algorithm. 
Simulation results are presented. The EADF framework is also 
tested with a standard dataset. 

Index Terms — Projection onto convex sets, active learning, 
decision fusion, online learning, entropy maximization, wildfire 
detection. 

I. Introduction 

AN online learning framework called Entropy functional 
based Adaptive Decision Fusion (EADF) is proposed 
which can be used for various image analysis and computer 
vision applications. In this framework, it is assumed that the 
compound algorithm consists of several sub-algorithms each 
of which yielding its own decision. The final decision is 
taken based on a set of real numbers representing confidence 
levels of various sub-algorithms. Decision values are linearly 
combined with weights which are updated online using an 
active fusion method based on performing entropic projections 
(e-projections) onto convex sets describing sub-algorithms. 

Adaptive learning methods based on orthogonal projections 
are successfully used in some computer vision and pattern 
recognition problems |fl~), El- In this active learning approach 



decisions from different classifiers are combined using a linear 
combiner Q. A multiple classifier system can prove useful for 
difficult pattern recognition problems especially when large 
class sets and noisy data are involved, because it allows the use 
of arbitrary feature descriptors and classification procedures 
at the same time J4). Instead of determining the weights 
using orthogonal projections as in (T), 0, we introduce the 
entropic e-projection approach which is based on a generalized 
projection onto convex set. 

The studies in the field of collective recognition, which were 
started in the middle of the 1950s, found wide application in 
practice during the last decade, leading to solution to complex 
large-scale applied problems {5). One of the first examples 
of the use multiple classifiers was given by Dasarathy in [3] 
in which he introduced the concept of composite classifier 
systems as a means of achieving improved recognition system 
performance compared to employing the classifier components 
individually. The method is illustrated by studying the case of 
the linear/NN(Nearest Neighbor) classifier composite system. 
Kumar and Zhang used multiple classifiers for palmprint 
recognition by characterizing the user's identity through the 
simultaneous use of three major palmprint representations and 
achieve better performance than either one individually |6|. A 
multiple classifier fusion algorithm is proposed for developing 
an effective video-based face recognition method Q. Garcia 
and Puig present results showing that pixel-based texture 
classification can be significantly improved by integrating 
texture methods from multiple families, each evaluated over 
multisized windows |8|. This technique consists of an initial 
training stage that evaluates the behavior of each considered 
texture method when applied to the given texture patterns of 
interest over various evaluation windows of different size. 

In this article, the EADF framework is applied to a computer 
vision based wildfire detection problem. The system based 
on this method is currently being used in more than 50 
forest fire lookout towers. The proposed automatic video 
based wildfire detection algorithm is based on five sub- 
algorithms: (i) slow moving video object detection, (ii) smoke- 
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colored region detection, (iii) wavelet transform based region 
smoothness detection, (iv) shadow detection and elimination, 
(v) covariance matrix based classification. Each sub-algorithm 
decides on the existence of smoke in the viewing range 
of the camera separately. Decisions from sub-algorithms are 
combined together by the adaptive decision fusion method. 
Initial weights of the sub-algorithms are determined from 
actual forest fire videos and test fires. They are updated by 
using entropic e-projections onto hyperplanes defined by the 
fusion weights. It is assumed that there is an oracle monitoring 
the decisions of the combined algorithm. In the wildfire 
detection case, the oracle is the security guard. Whenever a 
fire is detected the decision should be acknowledged by the 
security guard. The decision algorithm will also produce false 
alarms in practice. Whenever an alarm occurs the system asks 
the security guard to verify its decision. If it is incorrect the 
weights are updated according to the decision of the security 
guard. The goal of the system is not to replace the security 
guard but to provide a supporting tool to help him or her. The 
attention span of a typical security guard is only 20 minutes 
in monitoring stations. It is also possible to use feedback at 
specified intervals and run the algorithm autonomously at other 
times. For example, the weights can be updated when there is 
no fire in the viewing range of the camera and then the system 
can be run without feedback. 

The paper is organized as follows: Entropy functional based 
Adaptive Decision Fusion (EADF) framework is described 
in Section HI] The first part of the section describes our 
previous weight update algorithm which is obtained by or- 
thogonal projections onto convex sets Q], the second part 
proposes entropy based e-projection method for weight update 
of the sub-algorithms. Section HIT1 introduces the video based 
wildfire detection problem. The proposed framework is not 
restricted to the wildfire detection problem. It can be also 
used in other real-time intelligent video analysis applications 
in which a security guard is available. In Section [IV] each 
one of the five sub- algorithms which make up the compound 
(main) wildfire detection algorithm is described. In Section IVl 
experimental results are presented and the proposed online 
active fusion method is compared with the universal linear 
predictor and the weighted majority algorithms. The proposed 
EADF method is also evaluated on a dataset from the UCI 
machine learning repository (9]. Well-known classifiers (SVM, 
K-NN) are combined using EADF. During the training stage 
individual decisions of classifiers are used to find the weight 
of each classifier in the composite EADF classifier. Finally, 
conclusions are drawn in Section [VI] 

II. Adaptive Decision Fusion (ADF) Framework 

Let the compound algorithm be composed of ill-many de- 
tection sub-algorithms: D\, ...,Dm- Upon receiving a sample 
input x at time step n, each sub-algorithm yields a decision 
value Di(x,n) G R centered around zero. If Di(x,n) > 0, 
it means that the event is detected by the i-th sub-algorithm. 
Otherwise, it is assumed that the event did not happen. The 
type of the sample input x may vary depending on the algo- 
rithm. It may be an individual pixel, or an image region, or the 



entire image depending on the sub-algorithm of the computer 
vision problem. For example, in the wildfire detection problem 
presented in Section [TTH the number of sub-algorithms is M=5 
and each pixel at the location x of incoming image frame is 
considered as a sample input for every detection algorithm. 

Let D(x,n) = [D\(x, n), Dm{x, n)] T , be the vector of 
decision values of the sub-algorithms for the pixel at location 
x of input image frame at time step n, and w(x, n) = 
[w±(x, n), wm(x, n)] T be the current weight vector. For 
simplicity we will drop x in w(as, n) for the rest of the paper. 

We define 

y(x, n) = ~D T (x, n)w(n) = w l (n)D i (x, n) (1) 

i 

as an estimate of the correct classification result y(x, n) of the 
oracle for the pixel at location x of input image frame at time 
step n, and the error e(x, n) as e(x, n) — y(x, n)—y(x, n). As 
it can be seen in the next subsection, the main advantage of the 
proposed algorithm compared to other related methods in iflOl - 
lfl3ll . is the controlled feedback mechanism based on the error 
term. Weights of the algorithms producing incorrect (correct) 
decision is reduced (increased) iteratively at each time step. 
Another advantage of the proposed algorithm is that it does 
not assume any specific probability distribution about the data. 

A. Set Theoretic Weight Update Algorithm based on Orthog- 
onal Projections 

In this subsection, we first review the orthogonal projection 
based weight update scheme JT|. Ideally, weighted decision 
values of sub-algorithms should be equal to the decision value 
of y{x,n) the oracle: 

y(x, n) — T) T (x, n)w (2) 

which represents a hyperplane in the M-dimensional space, 
M. M . Hyperplanes are closed and convex in R A/ . At time 
instant n, T> T (x, n)w(n) may not be equal to y(x,n). In 
our approach, the next set of weights are determined by 
projecting the current weight vector w(n) onto the hyperplane 
represented by Eq. |2] The orthogonal projection w(n + 1) 
of the vector of weights w(n) € R M onto the hyperplane 
y(x,n) = T) T (x,n)w is the closest vector on the hyperplane 
to the vector w(n). 

Let us formulate the problem as a minimization problem: 

min| | w* -w(n)|| 

(3) 

subject to T> T (x, n)w* =y{x,n) 

The solution can be obtained by using Lagrange multipliers. 
If we define the next set of weights as w(n + 1) = w* it can 
be obtained by the following iteration: 

w(n + 1) = w(n) + ^D(;r, n) (4) 

where the Lagrange multiplier, A, can be obtained from the 
hyperplane equation: 

D T (a;,n)w* - y{x,n) = (5) 
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as follows: 

= y(x,n) -y(x,n) = e(x, n) 

||D(x,n)|| 2 ||D(.T,n)|| 2 K ' 

where the error, e(x, n), is defined as e(x, n) = y(x,n) — 
y(x, n) and y(x, n) = D T (x, n)w(n). Plugging this into Eq.|4] 

w(n + l)=w(n)+ ||D e ( ( ^ ) ) ||2 D(.x,n) (7) 

is obtained. Hence the projection vector is calculated according 
to Eq. 

Whenever a new input arrives, another hyperplane based on 
the new decision values D(x, n) of sub-algorithms, is defined 
in R M 

y(x,n+l) =~D T (x,n + l)w* (8) 

This hyperplane will not be the same as y(x, n) = 
D T (x, n)w(n) hyperplane. The next set of weights, w(n + 2), 
are determined by projecting w(?i + 1) onto the hyperplane in 
Eq. [8] Iterated weights converge to the intersection of hyper- 
planes fl4l . fl31 . The rate of convergence can be adjusted by 
introducing a relaxation parameter u to Eq. [7] as follows 

6 ( X TL) 

w(n + l)=w(n)+/x — ) ' * I>(x,n) (9) 

where < /i < 2 should be satisfied to guarantee the conver- 
gence according to the projections onto convex sets (POCS) 
theory lT6l-lT9t 

If the intersection of hyperplanes is an empty set, then 
the updated weight vector simply satisfies the last hyper- 
plane equation. In other words, it tracks decisions of the 
oracle by assigning proper weights to the individual sub- 
algorithms ifTTl . Ifl8l . 

The relation between support vector machines and orthog- 
onal projections onto halfplanes was established in lfT8l . Il20l 
and 1211 . As pointed out in 112011 SVM is very successful 
in batch settings but it cannot handle online problems with 
drifting concepts in which the data arrive sequentially. 



Algorithm 1 The pseudo-code for the POCS based algorithm 
Adaptive Decision Fusion(x,n) 
for i = 1 to M do 

Wi(0) = jj , Initialization 
end for 

e(x,n) = y(x,n) ~ y(x,n) 
for i = 1 to M do 

Wi{n + 1) <- Wi(n) + M jjn^jyp A(x,rt) 
end for 

n) = J2i Wi(n)Di(x, n) 
if y(x, n) > then 

return 1 
else 

return -1 
end if 



B. Entropic Projection (E-Projection) Based Weight Update 
Algorithm 

The l\ norm based minimization approaches provide suc- 
cessful signal reconstruction results in compressive sensing 
problems EH, E3, J24), E3. However the and h norm 
based cost functions used in compressive sensing problems are 
not differentiable everywhere. The entropy functional approx- 
imates the li norm J^i \ w i( n )\ f° r Wi(n) > B6I . Therefore 
it can be used to find approximate solutions to the inverse 
problems defined in |22l . Il23ll and other application requiring 

11 norm minimization. Bregman developed convex optimiza- 
tion algorithms in 1960's and his algorithms are widely used 
in many signal reconstruction and inverse problems lfl6l . 1271 . 
EE El, GO), US, ED, ED, E2, El- Bregman's method 
provides globally convergent iterative algorithms for problems 
with convex, continuous and differentiable cost functionals 
.9(0: 

min ,g(w) (10) 
wee 

such that 

T> T (x,n)w(n) = y for each time index n (11) 

In the EADF framework the cost function is g(w) = 
Y^f* Wi(n)log(wi(nj) and each equation in (fTTT > represents 
a hyperplane H n £ M. M which are closed and convex sets. 
In Bregman's method the iterative algorithm starts with an 
arbitrary initial estimate and successive e-projections are per- 
formed onto the hyperplanes H ni n = 1, 2, N in each step 
of the iterative algorithm. 

The e-projection onto a closed and convex set is a gen- 
eralized version of the orthogonal projection onto a convex 
set l26l . Let w(n) denote the weight vector for the nth sample. 
Its e-Projection w* onto a closed convex set C with respect 
to a cost functional <?(w) is defined as follows 

w* = arg min L(w,w(n)) (12) 

where 

L(w, w(n)) = <?(w) — g(w(n))— < V<?( w )i w — w(n) > 

(13) 

In the adaptive learning problem, we have the hyperplane H : 
D T (x, n).w(n + 1) = y(x, n) for each sample x. For each 
hyperplane H, the e-projection ( fT2b is equivalent to 

V fl(w(n + 1)) = V.9(w(n)) + AD(x, n) (14) 
T) T (x,n).w(n + 1) = y(x, n) (15) 

where A is the Lagrange multiplier. As pointed above the e- 
projection is a generalization of the orthogonal projection. 
When the cost functional is the Euclidean cost functional 
g(w) = X^ w *( n ) 2 me distance L(wi,W2) becomes the 

12 norm square of the difference vector (wi — W2), and 
the e-projection simply becomes the well-known orthogonal 
projection onto a hyperplane. 

When the cost functional is the entropy functional g(w) = 
Y2i w i( n ) \°§>( w i( n ))> me e-projection onto the hyperplane H 
leads to the following update equations: 

w t (n + l) =w l {n)e XD > (x > n \ * = 1,2, .... M (16) 
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where the Lagrange multiplier A is obtained by inserting ( fT6] > 
into the hyperplane equation: 

D T (a;, n)w(n + 1) = y(x,n) (17) 

because the e-projection w(n + 1) must be on the hyperplane 
H in Eq. Q3] This globally convergent iterative process is 
depicted in Fig. \T\ 
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,^y(x,n + 1) = D T (x.n + l)w* 



Fig. 1. Geometric interpretation of the entropic projection method: Weight 
vectors corresponding to decision functions at each frame are updated as to 
satisfy the hyperplane equations defined by the oracle's decision y(x,n) and 
the decision vector D(x, n). Lines in the figure represent hyperplanes in R M . 
Weight update vectors converge to the intersection of the hyperplanes. Notice 
that e-projections are not orthogonal projections. 

The above set of equations are used in signal reconstruc- 
tion from Fourier Transform samples and the tomographic 
reconstruction problem l27l . ifTTl . The entropy functional is 
defined only for positive real numbers which coincides with 
our positive weight assumption. 

The pseudo-code for the e-projection based adaptive deci- 
sion fusion based algorithm is given in Algorithm |2] To find 
the A value that minimizes the squared error at each iteration 
a simple search between possible A values can be performed 
or a nonlinear equation should be solved (Eqs. [T6land[T7]i. 



Algorithm 2 The pseudo-code for the EADF algorithm 
E-Projection Based Adaptive Decision Fusion(x,n) 
for i = 1 to M do 

Wi(0) = jj , Initialization 
end for 

for A = A mm to X max do 
for i = 1 to M do 

Vi (n) = Wi (n) 
Vi{n + 1) <- Vi {n)e XD ^ x ^ 
end for 

if \\y(x,n) - J2i v i( n + l)A(a:,n)|| < \\y(x,n) - 

T,i v i( n ) D i( x ,n)\\ then 

w T (n + 1) <- v(n + 1) 
end if 
end for 

w(n + 1) <— wt(b + 1) 
y(x,n) = J2i w i( n ) D i( x > n ) 
if y(x, n) > then 

return 1 
else 

return -1 
end if 



III. An Application: Computer Vision Based 
Wildfire Detection 

The Entropy function based Adaptive Decision Fusion 
(EADF) framework described in detail in the previous section 
with tracking capability is especially useful when the online 
active learning problem is of dynamic nature with drifting 
concepts Il33l - ll35l . In video based wildfire detection problem 
introduced in this section, the nature of forestal recordings vary 
over time due to weather conditions and changes in illumina- 
tion which makes it necessary to deploy an adaptive wildfire 
detection system. It is not feasible to develop one strong fusion 
model with fixed weights in this setting with drifting nature. 
An ideal online active learning mechanism should keep track 
of drifts in video and adapt itself accordingly. The projections 
in Eq. [16] and Eq. [7] adjust the importance of individual sub- 
algorithms by updating the weights according to the decisions 
of the oracle. 

Manned lookout posts are widely available in forests all 
around the world to detect wild fires. Surveillance cameras 
can be placed in these surveillance towers to monitor the 
surrounding forestal area for possible wild fires. Furthermore, 
they can be used to monitor the progress of the fire from 
remote centers. 

As an application of EADF, a computer vision based method 
for wildfire detection is presented in this article. Security 
guards have to work 24 hours in remote locations under 
difficult circumstances. They may simply get tired or leave the 
lookout tower for various reasons. Therefore, computer vision 
based video analysis systems capable of producing automatic 
fire alarms are necessary to help the security guards to reduce 
the average forest fire detection time. 

Cameras, once installed, operate at forest watch towers 
throughout the fire season for about six months which is 
mostly dry and sunny in Mediterranean region. There is 
usually a guard in charge of the cameras, as well. The 
guard can supply feed-back to the detection algorithm after 
the installation of the system. Whenever an alarm is issued, 
she/he can verify it or reject it. In this way, she/he can 
participate to the learning process of the adaptive algorithm. 
The proposed active fusion algorithm can be also used in other 
supervised learning problems where classifiers combinations 
through feedback is required. 

As described in the following section, the main wildfire 
detection algorithm is composed of five sub-algorithms. Each 
algorithm has its own decision function yielding a zero-mean 
real number for slow moving regions at every image frame 
of a video sequence. Decision values from sub-algorithms 
are linearly combined and weights of sub-algorithms are 
adaptively updated in our approach. 

There are several approaches on automatic forest fire de- 
tection in the literature. Some of the approaches are directed 
towards detection of the flames using infra-red and/or visible- 
range cameras and some others aim at detecting the smoke due 
to wildfire Il36l - ll40l . There are recent papers on sensor based 
fire detection BTI - 1431 . Infrared cameras and sensor based 
systems have the ability to capture the rise in temperature 
however they are much more expensive compared to regular 
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pan-tilt-zoom (PTZ) cameras. An intelligent space framework 
is described for indoor fire detection in fl4l . However, in 
this paper, an outdoor (forest) wildfire detection method is 
proposed. 

It is almost impossible to view flames of a wildfire from a 
camera mounted on a forest watch tower unless the fire is very 
near to the tower. However, smoke rising up in the forest due 
to a fire is usually visible from long distances. A snapshot of 
a typical wildfire smoke captured by a lookout tower camera 
from a distance of 5 km is shown in Fig. [2] 

Guillemant and Vicente l40l based their method on the 
observation that the movements of various patterns like smoke 
plumes produce correlated temporal segments of gray-level 
pixels. They utilized fractal indexing using a space-filling 
Z-curve concept along with instantaneous and cumulative 
velocity histograms for possible smoke regions. They made 
smoke decisions about the existence of smoke according to 
the standard deviation, minimum average energy, and shape 
and smoothness of these histograms. It is possible to include 
most of the currently available methods as sub-algorithms in 
the proposed framework and combine their decisions using the 
proposed EADF method. 




Fig. 2. Snapshot of a typical wildfire smoke captured by a forest watch tower 
which is 5 km away from the fire (rising smoke is marked with an arrow). 



IV. Building Blocks of Wildfire Detection 
Algorithm 

Wildfire detection algorithm is developed to recognize the 
existence of wildfire smoke within the viewing range of the 
camera monitoring forestal areas. The proposed wildfire smoke 
detection algorithm consists of five main sub-algorithms: 
(i) slow moving object detection in video, (ii) smoke-colored 
region detection, (iii) wavelet transform based region smooth- 
ness detection, (iv) shadow detection and elimination, (v) co- 
variance matrix based classification, with decision functions, 
Di(x,n), D2(x,n), Dj,(x,n), D±{x,n) and D^(x,n), re- 
spectively, for each pixel at location x of every incoming 
image frame at time step n. Computationally efficient sub- 
algorithms are selected in order to realize a real-time wildfire 
detection system working in a standard PC. The decision 
functions are combined in a linear manner and the weights 
are determined according to the weight update mechanism 
described in Section HT1 

Decision functions Di, i = 1,...,M of sub-algorithms 
do not produce binary values 1 (correct) or —1 (false), but 
they produce real numbers centered around zero for each 
incoming sample x. If the number is positive (negative), then 
the individual algorithm decides that there is (not) smoke due 
to forest fire in the viewing range of the camera. Output values 
of decision functions express the confidence level of each sub- 
algorithm. Higher the value, the more confident the algorithm. 

First four sub-algorithms are described in detail in ll52l 
which is available online at EURASIP webpage. We recently 
added the fifth sub-algorithm to our system. It is briefly 
reviewed below. 

A. Covariance Matrix Based Region Classification 

The fifth sub-algorithm deals with the classification of 
the smoke colored moving regions. A region covariance ma- 
trix l53l consisting of discriminative features is calculated for 
each region. For each pixel in the region, a 9-dimensional 
feature vector Zk is calculated as follows: 



Zk 



x 1 x 2 Y(xi,x 2 ) U(xi,x 2 ) V(xi : x 2 ) 



(18) 



dY(xi,x 2 ) 



dY(xi,x 2 ) 
dx 2 



d 2 Y{x u x 2 ) 
dx\ 



d 2 Y(xi 7 x 2 ) 



dx\ 



Smoke at far distances (> 100 m to the camera) exhibits 
different spatio-temporal characteristics than nearby smoke 
and fire I45l - ll47l . This demands specific methods explicitly 
developed for smoke detection at far distances rather than 
using nearby smoke detection methods described in l48l . 
The proposed approach is in accordance with the 'weak' 
Artificial Intelligence (AI) framework [49| introduced by Hu- 
bert L. Dreyfus as opposed to 'generalized' AI. According 
to this framework each specific problem in AI should be 
addressed as an individual engineering problem with its own 
characteristics ||50l . ||5T). 



where k is the label of a pixel, (xi,x 2 ) is the location of 
the pixel, Y, U, V are the components of the representation of 
the pixel in YUV color space, dY ^ 1,X2 ^ and dY ^ 1,x ^ are the 

1 1 ' ax\ dX2 

horizontal and vertical derivatives of the region respectively, 
calculated using the filter [-10 1], ^ Y{ ^ ] and d Y[x ^ X2) 
are the horizontal and vertical second derivatives of the region 
calculated using the filter [-1 2 -1], respectively. 

The feature vector for each pixel can be represented as 
follows: 

z k = [z k (i)} T (19) 



where, Zfe(i) is the i t h entry of the feature vector. This feature 
vector is used to calculate the 9 by 9 covariance matrix 
of the regions using the fast covariance matrix computation 
formula ll54l : 



Cr = [cr(i, j)] = 



1 



n- 1 



z k (i)zk(j) - Z kk 



(20) 



where 



^ n n 

Z k k = - Y] Zk(i) V z k (j) 



fe=l fc=l 

and n is the total number of pixels in the region and cn(i,j) 
is the the component of the covariance matrix. 

The region covariance matrices are symmetric therefore 
we only need half of the elements of the matrix for 
classification. We also do not need the first 3 elements 
cr(1, 1), cr(2, 1), cr(2, 2) when using the lower diagonal ele- 
ments of the matrix, because these are the same for all regions. 
Then, we need a feature vector fu with 9x 10/2 — 3 = 42 
elements for each region. For a given region the final feature 
vector does not depend on the number of pixels in the region, 
it only depends on the number of features in Zk- 

TABLE I 

Confusion matrix of the training set 





Predicted Labels 


Not Smoke 


Smoke 


Actual 
Labels 


Not Smoke 


11342/(97.2)% 


327/ (3.8%) 


Smoke 


49/ (0.7%) 


6962/(99.3%) 



A Support Vector Machine (SVM) with RBF kernel is 
trained with the region covariance feature vectors of smoke 
regions in the training database. 18680 images are used to 
train the SVM. 7011 of the images are positive images which 
have actual smoke and the rest are negative images that do not 
have smoke. Sample positive and negative images are shown 
in Fig. |3] The confusion matrix for the training set is given 
in Table U The success rate is 99.3% for the positive images 
and 97.2% for the negative images. 

The LIBSVM 155) software library is used to obtain the 
posterior class probabilities, pr = Pr(label = 1|/_r), where 
label — 1 corresponds to a smoke region. In this software 
library, posterior class probabilities are estimated by approxi- 
mating the posteriors with a sigmoid function as in ll56l . If the 
posterior probability is larger than 0.5 the label is 1 and the 
region contains smoke according to the covariance descriptor. 
The decision function for this sub-algorithm is defined as 
follows: 

D 5 (x,n) =2p R -l (21) 

where < pr < 1 is the estimated posterior probability 
that the region contains smoke. In ||53l , a distance measure 
based on eigenvalues are used to compare covariance matrices 
but we found that individual covariance values also provide 
satisfactory results in this problem. 

As pointed above decision results of five sub-algorithms, 
D%, D2, -D3, D4 and D$ are linearly combined to reach a 
final decision on a given pixel whether it is a pixel of a smoke 




(a) Negative training images. 




(b) Positive training images 
Fig. 3. Positive and negative images from the training set. 



region or not. Morphological operations are applied to the 
detected pixels to mark the smoke regions. The number of 
connected smoke pixels should be larger than a threshold to 
issue an alarm for the region. If a false alarm is issued during 
training phase, the oracle gives feedback to the algorithm by 
declaring a no-smoke decision value (y = —1) for the false 
alarm region. Initially, equal weights are assigned to each 
sub-algorithm. There may be large variations between forestal 
areas and substantial temporal changes may occur within the 
same forestal region. As a result weights of individual sub- 
algorithms will evolve in a dynamic manner over time. 

In real-time operating mode the PTZ cameras are in con- 
tinuous scan mode visiting predefined preset locations. In this 
mode constant monitoring from the oracle can be relaxed by 
adjusting the weights for each preset once and then using the 
same weights for successive classifications. Since the main 
issue is to reduce false alarms, the weights can be updated 
when there is no smoke in the viewing range of each preset 
and after that the system becomes autonomous. The cameras 
stop at each preset and run the detection algorithm for some 
time before moving to the next preset. By calculating separate 
weights for each preset we are able to reduce false alarms. 

V. Experimental Results 

A. Experiments on wildfire detection 

The proposed wildfire detection scheme with entropy func- 
tional based active learning method is implemented on a PC 
with an Intel Core Duo CPU 2.6GHz processor and tested 
with forest surveillance recordings captured from cameras 
mounted on top of forest watch towers near Antalya and Mugla 
provinces in Mediterranean region in Turkey. The weather is 
stable with sunny days throughout entire summer in Mediter- 
ranean. If it happens to rain there is no possibility of forest fire. 
The installed system successfully detected three forest fires in 
the summer of 2008. The system is also independently tested 
by the Regional Technology Clearing House of San Diego 
State University in California in April 2009 and it detected 
the test fire and did not produce any false alarms. A snapshot 
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from this test is presented in Fig. [4] It also detected another 
forest fire in Cyprus in 2010. The proposed EADF strategy is 



Algorithm 3 The pseudo-code for the universal predictor 




Fig. 4. A snapshot from an independent test of the system by the Regional 
Technology Clearing House of San Diego State University in California in 
April 2009. The system successfully detected the test fire and did not produce 
any false alarms. The detected smoke regions are marked with bounding 
rectangles. 



compared with the, projection onto convex sets (POCS) based 
algorithm and the universal linear predictor (ULP) scheme 
proposed by Singer and Feder {57] . The ULP adaptive filtering 
method is modified to the wildfire detection problem in an 
online learning framework. In the ULP scheme, decisions of 
individual algorithms are linearly combined similar to Eq. Q] 
as follows: 



Vu{x,n) 



Vi(n)Di(x,n) 



(22) 



where the weights, Uj(n), are updated according to the ULP 
algorithm, which assumes that the data (or decision values 
Di(x,n), in our case) are governed by some unknown prob- 
abilistic model P 1571 . The objective of a universal predictor 
is to minimize the expected cumulative loss. An explicit 
description of the weights, Vi(n), of the ULP algorithm is 
given as follows: 



Vi(n + 1) 



Ej exp(-^£(y(x,n),D j (x,n))) 



(23) 



where c is a normalization constant and the loss function for 
the i-th decision function is: 



£(y{x, n), Di(x, n)) = [y(x, n) - D^x, n)f (24) 

The constant c is taken as 4 as indicated in (57). The universal 
predictor based algorithm is summarized in Algorithm [3] 



Universal Predictor(x,n) 
for i = 1 to M do 

e(y(x,n),D l (x,n)) 



exp(- 



J2i ex P(-2E?(y(x,n),Dj(x,n))) 



Vi(n + 1) 
end for 
y u (x,n) =J2iVi(n)Di(x,n) 
if y u (x, n) > then 

return 1 
else 

return -1 
end if 



= [y(x, n) - Di(x, n)] 2 

-l(y(x,n),Di(x,n))) 



The POCS based scheme, the ULP based scheme, the 
EADF based scheme, and the non-adaptive approach with 
fixed weights are compared in the following experiments. In 
Tables [EI] and Hill 6-hour-long forest surveillance recordings 
containing actual forest fires and test fires as well as video 
sequences with no fires are used. 

We have 7 test fire videos ranging from 1 km to 4 km 
captured in Antalya and Mugla provinces in Mediterranean 
region in Turkey, in the summers of 2007 and 2008. To the 
best of our knowledge this is the largest database of forest 
fire clips having the initial stages of wildfires. The database 
is also used by the European Commission funded project 
FIRES ENSE l58l . All of the above mentioned decision fusion 
methods detect forest fires within 20 seconds, as shown in 
Table HI] The detection rates of the methods are comparable 
to each other. On the other hand, the proposed adaptive fusion 
strategy significantly reduces the false alarm rate of the system 
by integrating the feedback from the guard (oracle) into the 
decision mechanism within the active learning framework 
described in Section HI] In Fig. [5] a typical false alarm issued 
for moving tree leaves (which cause the white background 
to appear as moving smoke), by an untrained algorithm with 
decision weights equal to i is shown from the clip V\2. The 
proposed algorithm does not produce a false alarm in this 
video. 

The proposed method produces the lowest average error 
in our data set. A set of video clips containing moving 
cloud shadows and other moving regions that usually cause 
false alarms is used to generate Table [ill] These video clips 
are especially selected. The table shows the average pixel 
classification error for each method. The average pixel error 
for a video sequence v is calculated as follows: 



E(v) 



1 



F, 

-Y(-) 

n—1 



(25) 



where Nj is the total number of pixels in the image frame, 
Fi is the number of frames in the video sequence, e n is the 
sum of the squared errors for each classified pixel in image 
frame n. Except for one video sequence EADF based method 
has the lowest pixel classification error. 

In Fig. |6] the squared pixels errors of POCS and EADF 
based schemes are compared for the video clip V12. The 
weights are updated until 125th frame for both algorithms. 
The POCS based algorithm has an initial stage until 30th frame 
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Fig. 6. Average squared pixel errors for POCS and EADF based algorithms for the video seuqence V12. 




Fig. 5. False alarm from clip V12. Moving tree leaves in a forestal area 
cause a false alarm in an untrained algorithm with decision weights equal to 
i (depicted as a bounding box). The proposed algorithm does not produce a 
false alarm in this video. 



TABLE II 

Frame numbers at which an alarm is issued with different 
methods for wildfire smoke captured at various ranges and 
fps. it is assumed that the smoke starts at frame 0. 



Video 


Range 


Capture 


Frame number of first alarm 


Sequence 


(km) 


Frame Rate 


POCS 


Universal 


Fixed 


EADF 






(fps) 


Based 




Weights 


Based 


VI 


1 


7 


47 


48 


47 


42 


V2 


3 


7 


135 


142 


141 


125 


V3 


3 


7 


130 


101 


37 


134 


V4 


4 


25 


160 


154 


70 


150 


V5 


3 


9 


65 


55 


57 


56 


V6 


2 


5 


70 


74 


76 


75 


V7 


2 


5 


93 


74 


41 


83 


Average 






112.85 


92.57 


67 


97.85 



where the error gradually drops to zero, whereas EADF algo- 
rithm converges after only 2 frames. The tracking performance 
of the EADF algorithm is also better than the POCS based 
algorithm which can be observed after the frame number 180 
at which some of the sub-algorithms issue false alarms. 

The software is currently being used in 59 forest watch 
towers in Turkey. 

B. Experiments on a UCI Dataset 

The proposed method is also tested with a dataset from UCI 
(University of California, Irvine) machine learning repository 
to evaluate the performance of the algorithm in combining 
different classifiers. In the wildfire detection case the image 
data arrives sequentially and the decision weights are updated 



in real-time. On the other hand the UCI data sets are fixed. 
Therefore the dataset is divided into two parts. The first part 
is used for training. 

During the training phase, weights of different classifiers are 
determined using the EADF update method. In testing stage 
the fixed weights obtained from the training stage are used 
to combine the classifier decisions which process the data 
in a sequential manner because both the POCS and EADF 
frameworks assume that the new data arrive in a sequential 
manner. 

The test is performed on the ionosphere data from UCI 
machine learning repository that consists of radar measure- 
ments to detect the existence of free electrons that form a 
structure in the atmosphere. The electrons that show some kind 
of structure in the ionosphere return "Good" responses, the 
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TABLE III 

Average squared pixel errors issued by different methods to 
video sequences without any wildfire smoke. 



Video 
Sequence 


Frame 
Rate 
(fps) 


Video 
Duration 

(sec.) 


Average Errors (xlO ) 


POCS 
Based 


Universal 


Fixed 
Weights 


EADF 

Based 


V8 


7 


480 


7.0076 


94.6995 


138.2102 


9.3712 


V9 


25 


300 


8.3375 


38.2390 


54.9168 


5.5494 


V10 


25 


600 


8.9892 


77.5699 


101.7512 


3.2637 


Vll 


10 


900 


5.2054 


23.6602 


30.0455 


2.4314 


V12 


7 


60 


15.5350 


98.0371 


136.8163 


12.1520 


Average 






9.0149 


66.4411 


92.3480 


6.5535 



others return "Bad" responses. There are 351 samples with 34- 
element feature vectors that are obtained by passing the radar 
signals through an autocorrelation function. In ||59l , the first 
200 samples are used as training set to classify the remaining 
151 test samples. They obtained % 90.7 accuracy with a linear 
perceptron, % 92 accuracy with a non-linear perceptron, and 
% 96 accuracy with a back propagation neural network. 

For this test SVM, k-nn (k-Nearest Neighbor) and NCC 
(normalized cross-correlation) classifiers are used. Also, in this 
classification the decision functions of these classifiers produce 
binary values with 1 corresponding to "Good" classification 
and -1 corresponding to "Bad" classification rather than scaled 
posterior probabilities in the range [—1,1]. 

The accuracies of the sub-algorithms and EADF are shown 
in Table [TV] The success rates of the proposed EADF and 
POCS methods are both % 98.01 which is higher than all the 
sub-algorithms. Both the entropic projection and orthogonal 
projection based algorithms converge to a solution in the 
intersection of the convex sets. It turns out that they both 
converge to the same solution in this particular case. This is 
possible when the intersection set of convex sets is small. The 
proposed EADF method is actually developed for real-time 
application in which data arrives sequentially. This example is 
included to show that the EADF scheme can be also used in 
other datasets. It may be possible to get better classification 
results with other classifiers in this fixed UCI dataset. 

TABLE IV 

Accuracies of sub-algorithms and EADF on ionosphere 

DATASET. 



Data 


Success Rates (%) 


SVM 


k-nn (k=4) 


NCC 


POCS 


EADF 


Train 


100.0 


91.50 


100.0 


100.0 


100.0 


Test 


94.03 


97.35 


91.39 


98.01 


98.01 



VI. Conclusion 

An entropy functional based online adaptive decision fusion 
(EADF) is proposed for image analysis and computer vision 
applications with drifting concepts. In this framework, it 
is assumed that the main algorithm for a specific applica- 
tion is composed of several sub-algorithms each of which 
yielding its own decision as a real number centered around 
zero representing its confidence level. Decision values are 
linearly combined with weights which are updated online 
by performing non-orthogonal e-projections onto convex sets 
describing sub-algorithms. This general framework is applied 
to a real computer vision problem of wildfire detection. The 
proposed adaptive decision fusion strategy takes into account 
the feedback from guards of forest watch towers. Experimental 
results show that the learning duration is decreased with the 
proposed online adaptive fusion scheme. It is also observed 
that error rate of the proposed method is the lowest in our 
data set, compared to universal linear predictor (ULP) and the 
projection onto convex sets (POCS) based schemes. 

The proposed framework for decision fusion is suitable for 
problems with concept drift. At each stage of the algorithm, 
the method tracks the changes in the nature of the problem by 
performing an non-orthogonal e-projection onto a hyperplane 
describing the decision of the oracle. 
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